Bias Correction and Confidence Intervals for Fitted Q-iteration

We consider finite-horizon fitted Q-iteration with linear function approximation to learn a policy from a training set of trajectories. We show that fitted Q-iteration can give biased estimates and invalid confidence intervals for the parameters that feature in the policy. We propose a regularized estimator called soft-threshold estimator, derive it as an approximate empirical Bayes estimator, and show that it reduces bias and improves the coverage rates of confidence intervals via simulated experiments. We also demonstrate the use of this method in the analysis of data from a randomized smoking cessation study.